Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 test: retry GetOwnerGraph in owner references test on certificate errors #10201

Conversation

chrischdi
Copy link
Member

@chrischdi chrischdi commented Feb 27, 2024

What this PR does / why we need it:

When upgrading CAPI providers using clusterctl (as done in clusterctl upgrade tests):

  • We delete the provider except the CRD (to preserve existing CRs)
    • This includes the Certificate which is used for validating, mutating and conversion webhooks.
  • Afterwards we apply the provider again including a new Certificate CR
  • This leads to cert-manager issuing a new certificate
  • And cert-manager's ca-injector to inject the CA of the certificate into the ValidatingWebhookConfiguration, MutatingWebhookConfiguration and CRD (for the conversion webhook)

Injecting the CA into all CRDs takes some time. Our clusterctl upgrade tests immediately tries to query objects to e.g. verify that no unexpected rollouts happened.

This fails because querying objects might return from the kube-apiserver with the error x509: certificate signed by unknown authority when it tried to call the conversion webhook.

(Because the clusterctl upgrade introduced a new storageVersion (v1beta1) and kube-apiserver reads objects in their stored version from etcd (e.g. v1alpha4) and converts it to v1beta1 using the conversion webhook.)

The introduced code ensures that conversion webhooks work for all provider related CRDs by:

  • Listing all provider CRDs
  • Doing a list of the CRs of the CRD in all served versions

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Addresses flaky tests in: #9688

Note: testing at #10193

/area e2e-testing

@k8s-ci-robot k8s-ci-robot added area/e2e-testing Issues or PRs related to e2e testing cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 27, 2024
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 27, 2024
@chrischdi
Copy link
Member Author

/retitle 🌱 Wait for conversion-webhooks to be ready in clusterctl upgrade tests

@k8s-ci-robot k8s-ci-robot changed the title 🌱 implement waiting for conversion-webhooks to be ready in upgrade tests 🌱 Wait for conversion-webhooks to be ready in clusterctl upgrade tests Feb 27, 2024
@chrischdi
Copy link
Member Author

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 27, 2024
@chrischdi chrischdi force-pushed the pr-test-wait-for-certmanager-clusterctl-upgrade branch from 49a78c7 to 91b73e3 Compare February 27, 2024 13:15
@chrischdi
Copy link
Member Author

/test help

@k8s-ci-robot
Copy link
Contributor

@chrischdi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test pull-cluster-api-build-main
  • /test pull-cluster-api-e2e-blocking-main
  • /test pull-cluster-api-e2e-conformance-ci-latest-main
  • /test pull-cluster-api-e2e-conformance-main
  • /test pull-cluster-api-e2e-dualstack-and-ipv6-main
  • /test pull-cluster-api-e2e-main
  • /test pull-cluster-api-e2e-mink8s-main
  • /test pull-cluster-api-e2e-upgrade-1-29-1-30-main
  • /test pull-cluster-api-test-main
  • /test pull-cluster-api-test-mink8s-main
  • /test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

  • /test pull-cluster-api-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

  • pull-cluster-api-apidiff-main
  • pull-cluster-api-build-main
  • pull-cluster-api-e2e-blocking-main
  • pull-cluster-api-test-main
  • pull-cluster-api-verify-main

In response to this:

/test help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@chrischdi
Copy link
Member Author

/hold Need to check a failure occurence prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/10193/pull-cluster-api-e2e-blocking-release-1-5/1762437570274267136

/hold cancel

Changed to not use ListKind but Kind instead.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 27, 2024
@chrischdi
Copy link
Member Author

/test pull-cluster-api-e2e-main

@fabriziopandini
Copy link
Member

/lgtm
/approve

Feel free to unhold when you have finished validation on #10193

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 27, 2024
@fabriziopandini
Copy link
Member

/hold

@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 51c4f9b26a9ac2cc60c75e7cdbe600fac4b0db40

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Feb 27, 2024
@chrischdi
Copy link
Member Author

/retitle 🌱 [WIP] Wait for conversion-webhooks to be ready in clusterctl upgrade tests

/hold issue still occurs in tests 🤔

@k8s-ci-robot k8s-ci-robot changed the title 🌱 Wait for conversion-webhooks to be ready in clusterctl upgrade tests 🌱 [WIP] Wait for conversion-webhooks to be ready in clusterctl upgrade tests Feb 27, 2024
@chrischdi chrischdi force-pushed the pr-test-wait-for-certmanager-clusterctl-upgrade branch from 91b73e3 to dea02f4 Compare February 28, 2024 13:02
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 28, 2024
@k8s-ci-robot k8s-ci-robot removed the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 28, 2024
@k8s-ci-robot k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 28, 2024
@chrischdi
Copy link
Member Author

/retitle 🌱 test: retry GetOwnerGraph in owner references test on certificate errors

/hold issue still occurs in tests 🤔

@k8s-ci-robot k8s-ci-robot changed the title 🌱 [WIP] Wait for conversion-webhooks to be ready in clusterctl upgrade tests 🌱 test: retry GetOwnerGraph in owner references test on certificate errors Feb 28, 2024
@chrischdi chrischdi changed the title 🌱 test: retry GetOwnerGraph in owner references test on certificate errors 🐛 test: retry GetOwnerGraph in owner references test on certificate errors Feb 28, 2024
Copy link
Member

@fabriziopandini fabriziopandini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one!
/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 29, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: de86e79a78fade7ddaaadef0b6c2b7beb0104a07

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@chrischdi
Copy link
Member Author

/hold cancel

@chrischdi
Copy link
Member Author

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 29, 2024
@k8s-ci-robot k8s-ci-robot merged commit f85655a into kubernetes-sigs:main Feb 29, 2024
30 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.7 milestone Feb 29, 2024
@chrischdi
Copy link
Member Author

/cherry-pick release-1.6

@chrischdi
Copy link
Member Author

/cherry-pick release-1.5

@chrischdi
Copy link
Member Author

/cherry-pick release-1.4

@k8s-infra-cherrypick-robot

@chrischdi: #10201 failed to apply on top of branch "release-1.5":

Applying: test: retry GetOwnerGraph when having certificate issues
Using index info to reconstruct a base tree...
M	test/framework/ownerreference_helpers.go
Falling back to patching base and 3-way merge...
Auto-merging test/framework/ownerreference_helpers.go
CONFLICT (content): Merge conflict in test/framework/ownerreference_helpers.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 test: retry GetOwnerGraph when having certificate issues
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-infra-cherrypick-robot

@chrischdi: #10201 failed to apply on top of branch "release-1.4":

Applying: test: retry GetOwnerGraph when having certificate issues
Using index info to reconstruct a base tree...
M	test/framework/ownerreference_helpers.go
Falling back to patching base and 3-way merge...
Auto-merging test/framework/ownerreference_helpers.go
CONFLICT (content): Merge conflict in test/framework/ownerreference_helpers.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 test: retry GetOwnerGraph when having certificate issues
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-infra-cherrypick-robot

@chrischdi: new pull request created: #10217

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/e2e-testing Issues or PRs related to e2e testing cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants